Electronic device generating image data and converting the generated image data and operating method of the electronic device

ABSTRACT

An electronic device includes an image sensor configured to capture a target to generate first image data, and a processor configured to perform directional interpolation on a first area of the first image data to generate first partial image data, perform upscale on a second area of the first image data to generate second partial image data, and combine the first partial image data and the second partial image data to generate second image data.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority under 35 U.S.C. § 119 to Korean PatentApplication Nos. 10-2020-0030365 filed on Mar. 11, 2020 and10-2020-0086235 filed on Jul. 13, 2020, in the Korean IntellectualProperty Office, the disclosures of which are incorporated by referenceherein in their entireties.

TECHNICAL FIELD

Exemplary embodiments of the inventive concept described herein relateto an electronic device, and more particularly, relate to an electronicdevice generating image data and converting the generated image data.

DISCUSSION OF RELATED ART

An image sensor may generate image data indicating a target or sceneryfrom the target or the scenery. As performance of mobile devices such assmart phones and smart pads improve, image sensors may be employed inthe mobile devices. Image sensors employed in the mobile devices maygenerate image data and may be used to create image-based content.

One of the main functions for electronic devices generating image databy using image sensors is to generate image data having improvedresolution or image quality. Additionally, one of the main functions formobile electronic devices is to have low power consumption.

SUMMARY

According to an exemplary embodiment of the inventive concept, anelectronic device includes an image sensor configured to capture atarget to generate first image data, and a processor. The processor isconfigured to perform directional interpolation on a first area of thefirst image data to generate first partial image data, perform upscaleon a second area of the first image data to generate second partialimage data, and combine the first partial image data and the secondpartial image data to generate second image data.

According to an exemplary embodiment of the inventive concept, anoperating method of an electronic device which includes an image sensorand a processor includes capturing, at the image sensor, a target togenerate first image data, performing, at the processor, directionalinterpolation on a first area of the first image data to generate firstpartial image data, performing, at the processor, upscale on a secondarea of the first image data to generate second partial image data, andcombining, at the processor, the first partial image data and the secondpartial image data to generate second image data.

According to an exemplary embodiment of the inventive concept, anelectronic device includes an image sensor, and a processor configuredto receive first image data from the image sensor and convert and outputthe first image data as second image data. The image sensor includes alens, a color filter array that includes color filters configured topass specific frequency components of a light incident through the lens,a pixel array that includes pixels configured to convert intensities ofthe specific frequency components of the light passing through the colorfilters into analog signals, and an analog-to-digital converterconfigured to convert the analog signals into digital signals and outputthe digital signals to the image sensor. The processor includes a firstmemory configured to receive the first image data, location informationstorage configured to store location information, a first conversioncircuit configured to perform a first conversion on first input data andto output a result of the first conversion as first partial image data,a second conversion circuit configured to perform a second conversion onsecond input data and to output a result of the second conversion assecond partial image data, a selection circuit configured to output thefirst image data in the form of the first input data, the second inputdata, or the first input data and the second input data, based on thelocation information, and a mixer configured to combine the firstpartial image data and the second partial image data to generate thesecond image data, in response to a selection signal from the selectioncircuit. When the first partial image data and the second partial imagedata are output together, the mixer is configured to perform alphablending on the first partial image data and the second partial imagedata.

According to an exemplary embodiment of the inventive concept, anoperating method of an electronic device including an image sensor and aprocessor includes generating, at the image sensor, first image data,receiving, at the processor, partial data of the first image data,determining, at the processor, a location of the received partial databased on location information, where the location indicates whether thereceived partial data belongs to a first area, a second area, or a thirdarea, applying, at the processor, only a first conversion to thereceived partial data to generate converted partial data, when thereceived partial data belongs to the first area, applying, at theprocessor, only a second conversion to the received partial data togenerate the converted partial data, when the received partial databelongs to the second area, applying, at the processor, the firstconversion and the second conversion to the received partial data togenerate the converted partial data, when the received partial databelongs to the third area, and applying alpha blending on the convertedpartial data, when the received partial data belongs to the third area.The first conversion and the second conversion are different.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects and features of the inventive concept willbecome apparent by describing in detail exemplary embodiments thereofwith reference to the accompanying drawings.

FIG. 1 illustrates an electronic device according to an exemplaryembodiment of the inventive concept.

FIG. 2 illustrates an image sensor according to an exemplary embodimentof the inventive concept.

FIG. 3 illustrates an example in which color filters of a color filterarray of FIG. 1 are arranged depending on a first-type array patternaccording to an exemplary embodiment of the inventive concept.

FIG. 4 illustrates an example in which color filters of the color filterarray of FIG. 1 are arranged depending on a second-type array patternaccording to an exemplary embodiment of the inventive concept.

FIG. 5 illustrates an operating method of the electronic device of FIG.1 according to an exemplary embodiment of the inventive concept.

FIG. 6 illustrates a mixer according to an exemplary embodiment of theinventive concept.

FIG. 7 illustrates a calibration system for calibrating an image sensoraccording to an exemplary embodiment of the inventive concept.

FIG. 8 illustrates an operating method of the calibration system of FIG.7 according to an exemplary embodiment of the inventive concept.

FIG. 9 illustrates an example in which a calibration device of FIG. 7determines a first area, a second area, and a third area according to anexemplary embodiment of the inventive concept.

FIG. 10 illustrates a change of a modulation transfer function (MTF)according to a location on first image data according to an exemplaryembodiment of the inventive concept.

FIG. 11 illustrates an example in which a third area is divided intosub-areas according to an exemplary embodiment of the inventive concept.

FIG. 12 is a block diagram of an electronic device including amulti-camera module according to an exemplary embodiment of theinventive concept.

FIG. 13 is a detailed block diagram of a camera module of FIG. 12according to an exemplary embodiment of the inventive concept.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments of the inventive concept provide an electronicdevice generating image data having an improved resolution or imagequality and reducing power consumption, and an operating method of theelectronic device.

Below, exemplary embodiments of the inventive concept will be describedin detail with reference to the accompanying drawings. Like referencenumerals may refer to like elements throughout this application.

FIG. 1 illustrates an electronic device according to an exemplaryembodiment of the inventive concept. Referring to FIG. 1 , an electronicdevice 100 may include an image sensor 110 and a processor 120. Theimage sensor 110 may be based on a CMOS image sensor (CIS) or acharge-coupled device (CCD).

The image sensor 110 may include a lens 111, a color filter array 112, apixel array 113, and an analog-to-digital converter 114. The lens 111may transfer a light incident from the outside or a target to the colorfilter array 112.

The color filter array 112 may include color filters that are applied tolights incident through the lens 111. The color filter array 112 mayinclude color filters arranged in rows and columns. The color filtersmay include red filters “R”, green filters “G”, and blue filters “B”arranged according to a specific pattern.

The red filters “R” may pass a component (e.g., a frequency component)corresponding to a red color from among components of an incident light.The green filters “G” may pass a component corresponding to a greencolor from among the components of the incident light. The blue filters“B” may pass a component corresponding to a blue color from among thecomponents of the incident light.

The pixel array 113 may include pixels arranged in rows and columns. Thepixels of the pixel array 113 may correspond to the color filters of thecolor filter array 112, respectively.

For example, each of pixels corresponding to the green filters “G” mayoutput information corresponding to the amount (or intensity) of greenlight in the form of a current or voltage. Each of pixels correspondingto the red filters “R” may output information corresponding to theamount (or intensity) of red light in the form of a current or voltage.Each of pixels corresponding to the blue filters “B” may outputinformation corresponding to the amount (or intensity) of blue light inthe form of a current or voltage. Currents or voltages that the pixelsof the pixel array 113 output may be analog signals in which informationof light amounts (or intensities) are expressed by current amounts orvoltage levels.

The analog-to-digital converter 114 may convert the analog signalsoutput from the pixels of the pixel array 113 into digital signals. Thedigital signals converted by the analog-to-digital converter 114 may beoutput as first image data ID1. The first image data ID1 may correspondto one frame. The image sensor 110 may output the first image data ID1by obtaining image data in units of one or more rows and repeating theoutput of the obtained image data.

The processor 120 may include a first memory 121, location informationstorage 122, a selection block 123, a first conversion block 124, asecond conversion block 125, a mixer 126, and a second memory 127. Aswill be described further below, the selection block 123, the firstconversion block 124, the second conversion block 125, and the mixer 126may be circuits.

The first memory 121 may be configured to store the first image data ID1received from the image sensor 110. The first memory 121 may store thefirst image data ID1 corresponding to one frame by accumulating imagedata sequentially received from the image sensor 110. The first memory121 may include a random access memory (RAM). For example, the firstmemory 121 may include one of various random access memories such as adynamic RAM, a static RAM, a phase-change RAM, a ferroelectric RAM, amagnetic RAM, or a resistive RAM.

The location information storage 122 may store location information LI.The location information LI may be based on a characteristic of theimage sensor 110 (or the lens 111). For example, the locationinformation LI may include information about two or more areas, whichhave different resolutions (or image qualities), of the first image dataID1 obtained by the image sensor 110.

For example, due to characteristics of the image sensor 110 and due tocharacteristics and variables of components constituting the imagesensor 110, a resolution level of the first image data ID1 obtained bythe image sensor 110 may vary depending on a location on the first imagedata ID1. The location information LI may include information of aresolution of the first image data ID1, which is determined based on thelocation on the first image data ID1.

For example, the location information LI may include information of afirst area of the first image data ID1, in which a resolution (or imagequality) level is higher than a first threshold level, information of asecond area of the first image data ID1, in which a resolution (or imagequality) level is lower than or equal to a second threshold level, andinformation of a third area of the first image data ID1, in which aresolution (or image quality) level is lower than or equal to the firstthreshold level and higher than the second threshold level. In anexemplary embodiment of the inventive concept, the location informationLI may be obtained in calibration of the image sensor 110, and may bestored in the location information storage 122.

The location information storage 122 may include nonvolatile memorycells such as flash memory cells, phase-change memory cells,ferroelectric memory cells, resistive memory cells, or magnetic memorycells. The location information storage 122 may include an electricalfuse, an anti-fuse, or a laser fuse.

The selection block 123 may receive the location information LI from thelocation information storage 122. The selection block 123 may receivethe first image data ID1 from the first memory 121. The selection block123 may divide the first image data ID1 into first partial image dataID1 p 1 and second partial image data ID1 p 2 based on the locationinformation LI.

The selection block 123 may output partial data of the first image dataID1, which belong to the first area, as a portion of the first partialimage data ID1 p 1, based on the location information LI. The selectionblock 123 may output partial data of the first image data ID1, whichbelong to the second area, as a portion of the second partial image dataID1 p 2, based on the location information LI. The selection block 123may output partial data of the first image data ID1, which belong to thethird area, as a portion of the first partial image data ID1 p 1 and aportion of the second partial image data ID1 p 2, based on the locationinformation LI.

For example, the selection block 123 may receive partial data of thefirst image data ID1 in a specific unit. The specific unit may be in theform of one or more rows of the first image data ID1 or in the form of awindow. Depending on a location on the first image data ID1, at whichthe received partial data of the first image data ID1 is placed, theselection block 123 may activate at least one of the first conversionblock 124 and the second conversion block 125 based on the locationinformation LI.

The selection block 123 may output the received partial data of thefirst image data ID1 to a conversion block(s) activated from among thefirst conversion block 124 and the second conversion block 125.

For example, when the received partial data of the first image data ID1is determined based on the location information LI as being present inthe first area, the selection block 123 may activate the firstconversion block 124 and may output the received partial data of thefirst image data ID1 to the first conversion block 124 as a portion ofthe first partial image data ID1 p 1. The selection block 123 maygenerate and transmit a selection signal SEL to indicate that the firstconversion block 124 is selected.

When the received partial data of the first image data ID1 is determinedbased on the location information LI as being present in the secondarea, the selection block 123 may activate the second conversion block125 and may output the received partial data of the first image data ID1to the second conversion block 125 as a portion of the second partialimage data ID1 p 2. The selection block 123 may generate and transmitthe selection signal SEL to indicate that the second conversion block125 is selected.

When the received partial data of the first image data ID1 is determinedbased on the location information LI as being present in the third area,the selection block 123 may activate the first conversion block 124 andthe second conversion block 125, and may output the received partialdata of the first image data ID1 to the first conversion block 124 andthe second conversion block 125 as a portion of the first partial imagedata ID1 p 1 and a portion of the second partial image data ID1 p 2,respectively. The selection block 123 may generate and transmit theselection signal SEL to indicate that the first conversion block 124 andthe second conversion block 125 are selected.

The first conversion block 124 may perform a first conversion on thefirst partial image data ID1 p 1. The first conversion block 124 mayoutput a result of the conversion as a third partial image data ID2 p 1.The second conversion block 125 may perform a second conversion on thesecond partial image data ID1 p 2. The second conversion block 125 mayoutput a result of the conversion as a fourth partial image data ID2 p2.

For example, the first conversion may include remosaic, interpolation,or directional interpolation. Compared to the second conversion, thefirst conversion may use relatively complicated circuits and may consumea relatively great amount of power. Compared to the second conversion,the first conversion may maintain a quality of image data, inparticular, a high resolution.

For example, the second conversion may include binning or upscale. Thebinning may calculate an average or an intermediate value of pieces ofcolor information. Afterwards, a resolution may be improved based on theupscale. Compared to the first conversion, the second conversion may userelatively simple circuits and may consume relatively small amount ofpower.

The mixer 126 may generate second image data ID2 by combining the thirdpartial image data ID2 p 1 received from the first conversion block 124and the fourth partial image data ID2 p 2 received from the secondconversion block 125.

For example, when partial data of the first image data ID1 are convertedby the first conversion block 124, the mixer 126 may receive theconverted data as a portion of the third partial image data ID2 p 1 andmay store the converted data in the second memory 127. When partial dataof the first image data ID1 are converted by the second conversion block125, the mixer 126 may receive the converted data as a portion of thefourth partial image data ID2 p 2 and may store the converted data inthe second memory 127.

When partial data of the first image data ID1 are converted by the firstconversion block 124 and the second conversion block 125, the mixer 126may receive the converted data as a portion of the third partial imagedata ID2 p 1 and a portion of the fourth partial image data ID2 p 2, andmay store the converted data in the second memory 127. The mixer 126 mayperform alpha blending on the received pieces of data. The mixer 126 maystore a result of the alpha blending in the second memory 127.

In an exemplary embodiment of the inventive concept, the mixer 126 maystore a conversion result of partial data of the first image data ID1 atgiven locations of the second memory 127, such that the second imagedata ID2 are stored in the second memory 127 when the conversion of thefirst image data ID1 is completed.

The second memory 127 may include a random access memory (RAM). Forexample, the second memory 127 may include one of various random accessmemories such as a dynamic RAM, a static RAM, a phase-change RAM, aferroelectric RAM, a magnetic RAM, or a resistive RAM.

A physical block (PHY) 128 may output the second image data ID2 storedin the second memory 127 to an external device. For example, thephysical block 128 may output the second image data ID2 in units oflines. The physical block 128 may output the second image data ID2 basedon the C-PHY defined by the MIPI (Mobile Industry Processor Interface)specification.

In an exemplary embodiment of the inventive concept, the processor 120may be an application processor, a general-purpose processor, or aspecial-purpose image signal processor designed to process image data.Each of the components of the processor 120 may be implemented with ahardware circuit or may be implemented in the form of instructionsexecutable by a circuit configured to execute a command.

FIG. 2 illustrates an image sensor according to an exemplary embodimentof the inventive concept. An image sensor 200 of FIG. 2 may correspondto the image sensor 110 of FIG. 1 . Referring to FIGS. 1 and 2 , theimage sensor 200 may include a pixel array 210, a row driver 220, a rampsignal generator (RSG) 230, a comparator group 240, a counter group 250,a memory group 260, and a timing generator (TG) 270.

The pixel array 210 may include a plurality of pixels PX arranged alongrows and columns in the form of a matrix. Each of the plurality ofpixels PX may include a photoelectric conversion element. For example,the photoelectric conversion element may include a photo diode, a phototransistor, a photo gate, or a pinned photo diode. Each of the pluralityof pixels PX may sense a light by using the photoelectric conversionelement thereof, and may convert the amount of the sensed light into anelectrical signal, for example, a voltage or a current.

The plurality of pixels PX of the pixel array 210 may be divided into aplurality of pixel groups. Each pixel group may include at least two ormore pixels. In an exemplary embodiment of the inventive concept, apixel group may include pixels arranged in two rows and two columns orin three rows and three columns. Pixels constituting a pixel group mayshare at least one floating diffusion region.

The color filter array 112 including color filters respectivelycorresponding to the pixels PX of the pixel array 210 may be providedon/above the pixel array 210. The lens 111 may be provided on/above thecolor filter array 112. The color filters of the color filter array 112may include the red filters “R”, the green filters “G”, and the bluefilters “B”. A pixel group may correspond to color filters of the samecolor.

For example, a pixel group may include a red pixel to convert a light ofa red spectrum into an electrical signal, a green pixel to convert alight of a green spectrum into an electrical signal, or a blue pixel toconvert a light of a blue spectrum into an electrical signal, togetherwith a filter of the color filter array 112. For example, the filters ofthe color filter array 112 on the pixel array 210 may be arranged in theform of a Bayer pattern or in the form of a non-Bayer pattern.

The row driver 220 may be connected with rows of the pixels PX of thepixel array 210 through first to m-th row lines RL1 to RLm (where m is apositive integer). The row driver 220 may decode an address and/or acontrol signal generated by the timing generator 270. Depending on aresult of the decoding, the row driver 220 may sequentially select thefirst to m-th row lines RL1 to RLm of the pixel array 210, and may drivea selected row line with a specific voltage. For example, the row driver220 may drive a selected row line with a voltage appropriate for sensinga light.

Each of the first to m-th row lines RL1 to RLm connected with the rowsof the pixels PX may include two or more lines. The two or more linesmay include, for example, a signal for selecting a pixel, a signal forresetting a floating diffusion region, a signal for selecting a columnline, etc.

The ramp signal generator 230 may generate a ramp signal RS. The rampsignal generator 230 may operate under control of the timing generator270. For example, the ramp signal generator 230 may operate in responseto a control signal such as a ramp enable signal or a mode signal. Whenthe ramp enable signal is activated, the ramp signal generator 230 maygenerate the ramp signal RS having a slope set based on the mode signal.For example, the ramp signal generator 230 may generate the ramp signalRS that consistently decreases or increases from an initial level overtime.

The comparator group 240 may be connected with columns of the pixels PXof the pixel array 210 through first to n-th column lines CL1 to CLn(where n is a positive integer). The comparator group 240 may includefirst to n-th comparators C1 to Cn respectively connected with the firstto n-th column lines CL1 to CLn. The first to n-th comparators C1 to Cnmay receive the ramp signal RS from the ramp signal generator 230 incommon.

The first to n-th comparators C1 to Cn may compare voltages (orcurrents) of the first to n-th column lines CL1 to CLn with the rampsignal RS. When the ramp signal RS that consistently decreases (orincreases) becomes smaller (or greater) than voltages (or currents) ofthe first to n-th comparators C1 to Cn, each of the first to n-thcomparators C1 to Cn may invert an output signal. In other words, thefirst to n-th comparators C1 to Cn may output results of comparingmagnitudes (or amounts) of voltages (or currents), output from thepixels PX to the first to n-th column lines CL1 to CLn, with the rampsignal RS.

The counter group 250 may include first to n-th counters CNT1 to CNTnrespectively receiving output signals of the first to n-th comparatorsC1 to Cn. The first to n-th counters CNT1 to CNTn may start a countoperation at substantially the same time, for example, when, before, orafter the ramp signal RS starts to decrease (or increase). The first ton-th counters CNT1 to CNTn may stop the count operation when outputsignals of the first to n-th comparators C1 to Cn are inverted. Forexample, each of the first to n-th counters CNT1 to CNTn may stop thecount operation when an output signal of a corresponding comparatoramong the first to n-th comparators C1 to Cn is inverted.

In other words, the first to n-th comparators C1 to Cn may measuremagnitudes of voltages (or currents) of the first to n-th column linesCL1 to CLn by using the ramp signal RS, and the first to n-th countersCNT1 to CNTn may convert the measured results into digital values.

First to n-th memories M1 to Mn of the memory group 260 may includefirst to n-th memories M1 to Mn respectively receiving output signals ofthe first to n-th counters CNT1 to CNTn. The first to n-th memories M1to Mn may store the received output signals, and may output the storedsignals as the first image data ID1. For example, the first to n-thmemories M1 to Mn may include latches.

The timing generator 270 may control timings at which the image sensor200 operates. The timing generator 270 may control timings at which therow driver 220 sequentially selects the first to m-th row lines RL1 toRLm, and may control timings at which signals are transferred throughtwo or more lines included in a row line selected from the first to m-throw lines RL1 to RLm.

The timing generator 270 may control timings at which the ramp signalgenerator 230 generates the ramp signal RS and initializes the rampsignal RS. The timing generator 270 may control timings at which thefirst to n-th comparators C1 to Cn start a comparison operation and thefirst to n-th comparators C1 to Cn are initialized.

The timing generator 270 may control timings at which the first to n-thcounters CNT1 to CNTn start a count operation and the first to n-thcounters CNT1 to CNTn are initialized. The timing generator 270 maycontrol timings at which the first to n-th memories M1 to Mn output thefirst image data ID1 and the first to n-th memories M1 to Mn areinitialized.

According to an exemplary embodiment of the inventive concept, thetiming generator 270 may be configured to control various timings ofvarious components for the image sensor 200 to capture an image of atarget and to output the first image data ID1.

The row driver 220, the ramp signal generator 230, the comparator group240, the counter group 250, the memory group 260, and the timinggenerator 270 may correspond to the analog-to-digital converter 114converting analog signals generated by the pixels PX into digitalsignals.

In an exemplary embodiment of the inventive concept, the image sensor200 may generate and output image data in units of one row or two ormore rows of the pixels PX. The image sensor 200 may output the firstimage data ID1 corresponding to one frame by generating and outputtingthe image data while sequentially selecting the rows RL1 to RLm of thepixels PX.

FIG. 3 illustrates an example in which color filters of a color filterarray of FIG. 1 are arranged depending on a first-type array patternaccording to an exemplary embodiment of the inventive concept. Referringto FIGS. 1, 2, and 3 , the color filter array 112 may include threecolor filters, e.g., red filters “R”, green filters “G”, and bluefilters “B”. The color filters of the color filter array 112 may bearranged in rows and columns in units of a first basic unit BU1,depending on the first-type array pattern.

The first basic unit BU1 may include four color filters. The first basicunit BU1 may include the green filter “G”, the red filter “R”, the greenfilter “G”, and the blue filter “B” arranged sequentially in a clockwisedirection from the left top. In an exemplary embodiment of the inventiveconcept, the first-type array pattern of the color filter array 112 maybe a Bayer pattern.

Image data based on the Bayer pattern may be processed based on thefirst basic unit BU1. For example, an image based on the Bayer patternmay be converted into image data that is easy to process, for example,RGB data. In the conversion process, the first basic unit Bill may beused as one pixel data, and one R signal, one G signal, and one B signalmay be generated from the one first basic unit BU1.

The Bayer pattern has been used as an array pattern of color filterarrays of image sensors for a long time. Accordingly, processorsprocessing image data have been implemented to convert image data basedon the Bayer pattern into RGB data.

As technology for manufacturing image sensors develops, the resolutionof pixel arrays is increasing. As the resolution of pixel arraysincreases, color filter arrays may be implemented to include colorfilters arranged according to an array pattern other than the Bayerpattern.

FIG. 4 illustrates an example in which color filters of the color filterarray of FIG. 1 are arranged depending on a second-type array patternaccording to an exemplary embodiment of the inventive concept. Referringto FIGS. 1, 2, and 4 , the color filter array 112 may include threecolor filters, e.g., red filters “R”, green filters “G”, and bluefilters “B”. The color filters of the color filter array 112 may bearranged in rows and columns in units of a second basic unit BU2,according to the second-type array pattern.

The second basic unit BU2 may include 12 color filters. In the case ofequally dividing the second basic unit BU2 into four quadrants based ona horizontal axis and a vertical axis, the second basic unit BU2 mayinclude four G color filters placed at the upper left quadrant, four Rcolor filters placed at the upper right quadrant, four B color filtersplaced at the lower left quadrant, and four G color filters placed atthe lower right quadrant.

In an exemplary embodiment of the inventive concept, the second basicunit BU2 may include three or more color filters that are disposedadjacent to each other and correspond to the same color. The second-typearray pattern of the color filter array 112 may not be the Bayerpattern.

For example, the second-type array pattern may be a non-Bayer pattern.The color filter array 112 of the image sensor 110 may include colorfilters arranged based on the non-Bayer pattern as illustrated in FIG. 4. Accordingly, the first image data ID1 fails to be processed by ageneral processor based on the non-Bayer pattern.

The first conversion block 124 and the second conversion block 125 mayconvert the first image data ID1 based on the non-Bayer patternillustrated in FIG. 4 into the second image data ID2 based on the Bayerpattern illustrated in FIG. 3 . The first area of the first image dataID1 may be converted by the first conversion block 124. The second areaof the first image data ID1 may be converted by the second conversionblock 125.

The third area of the first image data ID1 may be converted by the firstconversion block 124 and the second conversion block 125. Through thefirst conversion block 124 and the second conversion block 125, thesecond image data ID2 based on the Bayer pattern corresponding to thefirst basic unit Bill may be generated from the first image data ID1based on the non-Bayer pattern corresponding to the second basic unitBU2.

A detailed example of the non-Bayer pattern is illustrated in FIG. 4 ,but an array pattern of the color filter array 112 is not limited to theexample illustrated in FIG. 4 . The array pattern of the color filterarray 112 may be implemented in various shapes including an arraypattern called “tetra” or “nona”.

FIG. 5 illustrates an operating method of the electronic device of FIG.1 according to an exemplary embodiment of the inventive concept.Referring to FIGS. 1 and 5 , in operation S110, the electronic device100 may generate the first image data ID1. For example, the image sensor110 may generate the first image data ID1 based on the non-Bayer patternand may store the first image data ID1 in the first memory 121. Thefirst image data ID1 may correspond to one frame.

In operation S120, the electronic device 100 may select partial data ofthe first image data ID1. For example, the selection block 123 of theprocessor 120 may receive partial data of the first image data ID1stored in the first memory 121, e.g., partial data of one frame.

In operation S130, the electronic device 100 may determine a location ofthe received partial data. For example, the selection block 123 of theprocessor 120 may determine whether the received partial data belong tothe first area, the second area, or the third area, based on thelocation information LI.

When the received partial data belong to the first area, the receivedpartial data may belong to a location of image data that has the highestresolution. The selection block 123 may output the received partial datato the first conversion block 124 as a portion of the first partialimage data ID1 p 1. In operation S140, the first conversion block 124may apply the first conversion to the received partial data. Theconverted partial data may be transferred to the mixer 126 as a portionof the third partial image data ID2 p 1. The mixer 126 may store theconverted partial data at a corresponding location of the second memory127. Afterwards, operation S180 may be performed, which will bedescribed below.

Returning to operation S130, when the received partial data belong tothe second area, the received partial data may belong to a location ofthe image data that has the lowest resolution. The selection block 123may output the received partial data to the second conversion block 125as a portion of the second partial image data ID1 p 2. In operationS150, the second conversion block 125 may apply the second conversion tothe received partial data. The converted partial data may be transferredto the mixer 126 as a portion of the fourth partial image data ID2 p 2.The mixer 126 may store the converted partial data at a correspondinglocation of the second memory 127. Afterwards, operation S180 may beperformed.

Returning to operation S130, when the received partial data belong tothe third area, the received partial data may belong to a location ofthe image data that has a resolution of an intermediate level. Theselection block 123 may output the received partial data to the firstconversion block 124 as a portion of the first partial image data ID1 p1 and may output the received partial data to the second conversionblock 125 as a portion of the second partial image data ID1 p 2. Inoperation S160, the first conversion block 124 may apply the firstconversion to the received partial data, and the second conversion block125 may apply the second conversion to the received partial data. Theconverted partial data may be transferred to the mixer 126 as a portionof the third partial image data ID2 p 1 and a portion of the fourthpartial image data ID2 p 2.

When the partial data belong to the third area, in operation S170following operation S160, the mixer 126 may perform the alpha blendingon the converted partial data (e.g., first converted partial data)output from the first conversion block 124 and the converted partialdata (e.g., second converted partial data) output from the secondconversion block 125.

For example, the mixer 126 may apply a first transparency to the firstconverted partial data and may apply a second transparency to the secondconverted partial data. The mixer 126 may blend the first convertedpartial data to which the first transparency is applied and the secondconverted partial data to which the second transparency is applied, andmay store the blended data at a corresponding location of the secondmemory 127.

In operation S180, the electronic device 100 may determine whether theconversion of last partial data of the image data is completed. When theconversion of last partial data of the image data is not completed, nextpartial data may be received in operation S120, and the conversiondescribed with reference to operation S130 to operation S170 may beperformed on the next partial data. When the conversion of the lastpartial data is completed, the electronic device 100 may terminate theconversion of the image data.

In other words, the processor 120 of the electronic device 100 mayperform only the first conversion on image data in the first area of thefirst image data ID1 corresponding to one frame, and may perform onlythe second conversion on image data in the second area of the firstimage data ID1. The processor 120 may perform both the first conversionand the second conversion on image data in the third area of the firstimage data ID1 and may perform the alpha blending on a result of thefirst conversion and a result of the second conversion. As describedabove, the first conversion may include remosaic, interpolation, ordirectional interpolation, and the second conversion may include binningor upscale. In other words, the first conversion and the secondconversion may be different.

In an exemplary embodiment of the inventive concept, the result of thesecond conversion may be used as background image data, and the resultof the first conversion may be used as foreground image data. As anotherexample, the result of the first conversion may be used as backgroundimage data, and the result of the second conversion may be used asforeground image data. The transparency of the foreground image data ofthe alpha blending, e.g., an alpha value, may be stored in the locationinformation storage 122 together with the location information LI. Thealpha value may be determined in a calibration of the image sensor 110.

FIG. 6 illustrates a mixer according to an exemplary embodiment of theinventive concept. A mixer 300 may correspond to the mixer 126 includedin the processor 120 of the electronic device 100 of FIG. 1 . Referringto FIGS. 1 and 6 , the mixer 300 may include a first multiplexer 310, asecond multiplexer 320, an alpha blender 330, and a combiner 340.

The first multiplexer 310 may receive the third partial image data ID2 p1. In response to the selection signal SEL, the first multiplexer 310may output partial data ID2A1 of the third partial image data ID2 p 1,which correspond to the first area, to the combiner 340, and may outputpartial data ID2A3 a of the third partial image data ID2 p 1, whichcorrespond to the third area, to the alpha blender 330.

The second multiplexer 320 may receive the fourth partial image data ID2p 2. In response to the selection signal SEL, the second multiplexer 320may output partial data ID2A2 of the fourth partial image data ID2 p 2,which correspond to the second area, to the combiner 340, and may outputpartial data ID2A3 b of the fourth partial image data ID2 p 2, whichcorrespond to the third area, to the alpha blender 330.

The alpha blender 330 may perform the alpha blending on the partial dataID2A3 a received from the first multiplexer 310 and the partial dataID2A3 b received from the second multiplexer 320. The alpha blender 330may output a result of the alpha blending as partial data ID2A3,corresponding to the third area, to the combiner 340.

The combiner 340 may receive the partial data ID2A1 corresponding to thefirst area from the first multiplexer 310 and may receive the partialdata ID2A2 corresponding to the second area from the second multiplexer320. The combiner 340 may receive the partial data ID2A3 correspondingto the third area from the alpha blender 330. The combiner 340 maycombine the received data (e.g., ID2A1, ID2A2, and ID2A3) to output thesecond image data ID2 in units of frames.

FIG. 7 illustrates a calibration system for calibrating the image sensorof FIG. 1 according to an exemplary embodiment of the inventive concept.Referring to a calibration system 400 illustrated in FIG. 7 , after theimage sensor 110 is manufactured, the image sensor 110 may be connectedwith a calibration device 420 for calibration. In calibration, acalibration sheet 410 may be photographed by using the image sensor 110.The calibration sheet 410 may include various patterns corresponding tovarious resolutions. The calibration device 420 may perform calibrationbased on a result of photographing the calibration sheet 410 by theimage sensor 110.

For example, the calibration may include calibrating a slope of the lens111 of the image sensor 110, and adjusting a distance between the lens111 and the pixel array 113 to calibrate a resolution. Additionally, thecalibration may include determining the first area, the second area, andthe third area of image data generated by the image sensor 110. Thecalibration device 420 may generate the location information LIincluding information of the first area, the second area, and the thirdarea.

The calibration may further include determining a transparency to beapplied to the third area, e.g., an alpha value. The locationinformation LI and the alpha value may be stored in the locationinformation storage 122 without modification or with subsequent revisionby a manufacturer of the electronic device 100 (refer to FIG. 1 ) (e.g.,based on a process characteristic or a product characteristic of themanufacturer).

FIG. 8 illustrates an operating method of the calibration system of FIG.7 according to an exemplary embodiment of the inventive concept.Referring to FIGS. 7 to 8 , in operation S210, the calibration system400 may generate image data of the calibration sheet 410 by using theimage sensor 110.

In operation S220, the calibration device 420 may calibrate the imagesensor 110 and may determine whether the calibration of the image sensor110 passes. For example, when a resolution (or an image quality) ofimage data generated by the image sensor 110 thus calibrated is higherthan or equal to a threshold value, the image sensor 110 may bedetermined as passing the calibration.

When it is determined that the image sensor 110 passes the calibration,in operation S230, the calibration device 420 may determine the firstarea, the second area, and the third area of first image data generatedby the image sensor 110, and may determine an alpha value to be appliedto the third area.

In operation S240, the calibration device 420 may output the locationinformation LI indicating the first area, the second area, and the thirdarea, and the alpha value. For example, when the image sensor 110 iscoupled to the processor 120, the location information LI and the alphavalue may be stored in the location information storage 122 of theprocessor 120. In the case where the image sensor 110 is provided to amanufacturer of the electronic device 100 without coupling to theprocessor 120, the location information LI and the alpha value may beprovided to the manufacturer together.

When it is determined in operation S220 that the image sensor 110 doesnot pass the calibration, operation S250 is performed. In operationS250, the calibration device 420 may determine a calibration fail or maydetermine the image sensor 110 as a recalibration target. When thecalibration fail is determined, the image sensor 110 may be discarded ormay be reused in a product providing a lower resolution (or imagequality). When the image sensor 110 is targeted for recalibration, therecalibration may be performed on the image sensor 110, or therecalibration may be reserved.

FIG. 9 illustrates an example in which a calibration device of FIG. 7determines a first area, a second area, and a third area according to anexemplary embodiment of the inventive concept. Referring to FIGS. 1, 7,and 9 , in operation S310, the calibration device 420 may detect amodulation transfer function (MTF) according to a location on the firstimage data ID1 of the image sensor 110.

In operation S320, the calibration device 420 may detect an area where achange of the MTF is greater than a threshold value. For example, thecalibration device 420 may detect an area where the variations in theMTF for a unit length according to a specific coordinate axis on thefirst image data ID1 are greater than the threshold value. As anotherexample, the calibration device 420 may detect an area where aninstantaneous rate of change of the MTF is greater than the thresholdvalue.

In operation S330, the calibration device 420 may determine the firstarea, the second area, and the third area based on the detected area.For example, the calibration device 420 may determine the detected areaas the third area. The calibration device 420 may determine an areahaving an MTF higher than an MTF of the detected area as the first area.The calibration device 420 may determine an area having an MTF lowerthan the MTF of the detected area as the second area.

FIG. 10 illustrates a change of an MTF according to a location on thefirst image data according to an exemplary embodiment of the inventiveconcept. Referring to FIGS. 1 and 10 , a location on the first imagedata ID1 may be expressed by a field. The field may indicate a distancefrom the center of the first image data ID1.

For example, the field may be a ratio of a distance from the center onthe first image data ID1 to a long side (or a short side) of the firstimage data ID1. In other words, the field indicating a location on thefirst image data ID1 may indicate a radial distance from the center.

A horizontal axis of a first graph G1 matched with the first image dataID1 indicates a field on the first image data ID1, and a vertical axisthereof indicates an MTF. For example, a unit of the MTF may be a cycleper pixel (cycle/pixel).

As illustrated in FIG. 10 , an area where a change of the MTF is greaterthan a threshold value may be defined as a third area A3. An area havingan MTF greater than the MTF of the third area A3 may be defined as afirst area A1. An area having an MTF smaller than the MTF of the thirdarea A3 may be defined as a second area A2.

As a curvature of the lens 111 increases and pitches of pixels of thepixel array 113 are smaller than an airy disk, a change of the MTFaccording to a location on the first image data ID1 increases.

The resolution and the quality of the second image data ID2 may beimproved by applying the first conversion block 124 having high-qualityand high-power characteristics to image data of the first area A1 havinga relatively high resolution.

Power consumption may be reduced in the process of generating the secondimage data ID2 by applying the second conversion block 125 havinglow-quality and low-power characteristics to image data of the secondarea A2 having a relatively low resolution.

In other words, the first area A1 and the second area A2 are differentareas having different resolutions. The first conversion performed bythe first conversion block 124 (e.g., remosaic, interpolation, ordirectional interpolation) generates image data having a resolutionhigher than that of image data generated by the second conversionperformed by the second conversion block 125 (e.g., binning or upscale).On the other hand, the second conversion consumes less power than thefirst conversion. As such, the electronic device 100 may adaptively orselectively perform the first conversion and the second conversiondepending on a location (e.g., different areas) on the first image dataID1 to generate the second image data ID2, which generates a higherresolution image as compared to when only performing the secondconversion and consumes less power as compared to when only performingthe first conversion. This will be described in further detail belowwith reference to Equations 1 to 3 and Tables 1 and 2.

With regard to image data of the third area A3 where a change of the MTFis great, as the alpha blending is applied to a conversion result of thefirst conversion block 124 and a conversion result of the secondconversion block 125, it may be possible to prevent an unintendedpattern from occurring at a boundary of the first area A1 and the secondarea A2 on the second image data ID2 and to prevent a resolution and animage quality from sharply changing.

As the field radially indicates a location on the first image data ID1,the first area A1 may be a circle, and the second area A2 and the thirdarea A3 may include a remaining area of the first image data ID1, whichsurrounds the first area A1. For example, the third area A3 may be aconcentric circle surrounding the first area A1, and the second area A2may be the remaining area surrounding the first area A1 and the thirdarea A3.

In an exemplary embodiment of the inventive concept, an alpha value tobe applied to the third area A3 may be determined by the calibrationdevice 420 based on various factors such as a value (e.g., anintermediate value, an average, or variations) of an MTF of the thirdarea A3, a relationship (e.g., a ratio) of the value of the MTF of thethird area A3 and a value (e.g., an intermediate value, an average, orvariations) of an MTF of the first area A1, and a relationship (e.g., aratio) of the value of the MTF of the third area A3 and a value (e.g.,an intermediate value, an average, or variations) of an MTF of thesecond area A2. For example, the calibration device 420 may beconfigured to determine an alpha value through a machine learning-basedinference on values of an MTF.

As another example, due to a characteristic of a spherical convex lens,a resolution of a central portion on the first image data ID1 may be thehighest. Accordingly, the central portion on the first image data ID1may be determined as the first area A1.

Likewise, due to the characteristic of the spherical convex lens, aresolution of an outer portion on the first image data ID1 may be thelowest. Accordingly, the outer portion on the first image data ID1 maybe determined as the second area A2. An area between the first area A1and the second area A2 may be determined as the third area A3. In thethird area A3, first converted partial data and second converted partialdata may be blended by the alpha blending.

When the first conversion block 124 and the second conversion block 125according to an exemplary embodiment of the inventive concept areadaptively used, power consumption may be expressed by Equation 1 below.P=A1·P1+A2·P2+A3(P1+P2)  [Equation 1]

In Equation 1 above, “A1” may be the area of the first area A1, “A2” maybe the area of the second area A2, and “A3” may be the area of the thirdarea A3. “P1” may be power consumption of the first conversion block124, and “P2” may be power consumption of the second conversion block125. In general, the power consumption of the second conversion block125 may correspond to about 10% of the power consumption of the firstconversion block 124. Accordingly, Equation 1 may be summarized as thefollowing Equation 2.P=A1·P1+A2·0.1P1+A3(P1+0.1P1)  [Equation 2]

Equation 2 above may be summarized as the following Equation 3.P=P1(A1+0.1A2+1.1A3)  [Equation 3]

Because the area of the second area A2 is the largest on the first imagedata ID1, power consumption may be reduced as much as about 48.3%compared to when only the first conversion is performed. Accordingly,the power consumption of the electronic device 100 may be reduced.

Table 1 below shows the MTF when the first conversion is performed, whenthe second conversion is performed, and when the first conversion andthe second conversion according to exemplary embodiments of theinventive concept are adaptively performed.

TABLE 1 First conversion and First conversion Second conversion secondconversion MTF50 MTF30 MTF10 MTF50 MTF30 MTF10 MTF50 MTF30 MTF10 Center0.2324 0.2819 0.3662 0.1845 0.2125 0.2460 0.2324 0.2819 0.3662 0.5 field0.1915 0.2776 0.3375 0.1541 0.1977 0.2424 0.1667 0.2370 0.2958 0.8 field0.1974 0.2148 0.2453 0.1874 0.1969 0.213  0.1874 0.1969 0.213 

From the information described in Table 1, a decrease in a resolution(or image quality) compared to when the first conversion is performed isindicated by Table 2 below.

TABLE 2 First conversion and Second conversion second conversion MTF50MTF30 MTF10 MTF50 MTF30 MTF10 Center 79.39% 75.38% 67.18% 100%   100%    100%    0.5 field 80.47% 71.82% 87.08%  87.08%  85.36%  87.65%0.8 field 94.93% 91.67% 86.83%  94.93%  91.67%  86.83%

As shown in Table 2 above, as the first conversion and the secondconversion according to exemplary embodiments of the inventive conceptare adaptively applied, the electronic device 100 generates the firstimage data ID1 having an improved resolution (or image quality) ascompared to when only the second conversion is applied, and reducespower consumption as compared to when only the first conversion isapplied.

The above numerical values may be used in an exemplary environment ofthe inventive concept and may vary depending on a size of the first areaA1, a size of the second area A2, a size of the third area A3, thenumber of sub-areas of the third area A3, sizes of the sub-areas of thethird area A3, and an alpha value to be applied to the third area A3.

FIG. 11 illustrates an example in which a third area is divided intosub-areas according to an exemplary embodiment of the inventive concept.Referring to FIGS. 1, 7, and 11 , the third area A3 may be divided intosub-areas depending on a distance from the center. Each of the sub-areasmay be in the form of a concentric circle surrounding the first area A1.

The sub-areas may be specified to have different alpha values. Forexample, as a location of a sub-area becomes closer to the first areaA1, an alpha value may decrease (or increase). As a location of asub-area becomes closer to the second area A2, an alpha value mayincrease (or decrease).

In other words, as an example, for one sub-area of the third area A3,the first conversion block 124 may perform the first conversion tooutput the third partial image data ID2 p 1, and the second conversionblock 125 may perform the second conversion to output the fourth partialimage data ID2 p, as described with reference to FIG. 1 . The alphablender 330 may perform the alpha blending on the partial data ID2A3 aof the third partial image data ID2 p land the partial data ID2A3 b ofthe fourth partial image data ID2 p 2 to output the partial data ID2A3(e.g., fifth partial image data), as described with reference to FIG. 6. For another sub-area of the third area A3, the first conversion block124 may perform the first conversion to output sixth partial image data,the second conversion block 125 may perform the second conversion tooutput to output seventh partial image data, and the alpha blender 330may perform the alpha blending on the sixth partial image data and theseventh partial image data to generate eighth partial image data. Inthis case, an alpha value applied to the fifth partial image data and analpha value applied to the eighth partial image data are different.

In an exemplary embodiment of the inventive concept, an output of thefirst conversion block 124 may be used as foreground image data, and anoutput of the second conversion block 125 may be used as backgroundimage data. In this case, as a location of a sub-area becomes closer tothe first area A1, an alpha value of the foreground image data maydecrease. As a location of a sub-area becomes closer to the second areaA2, an alpha value of the foreground image data may increase.

As another example, an output of the first conversion block 124 may beused as background image data, and an output of the second conversionblock 125 may be used as foreground image data. In this case, as alocation of a sub-area becomes closer to the first area A1, an alphavalue of the foreground image data may increase. As a location of asub-area becomes closer to the second area A2, an alpha value of theforeground image data may decrease.

FIG. 12 is a block diagram of an electronic device including amulti-camera module according to an exemplary embodiment of theinventive concept. FIG. 13 is a detailed block diagram of a cameramodule of FIG. 12 according to an exemplary embodiment of the inventiveconcept.

Referring to FIG. 12 , an electronic device 1000 may include a cameramodule group 1100, an application processor 1200, a power managementintegrated circuit (PMIC) 1300, and an external memory 1400.

The camera module group 1100 may include a plurality of camera modules1100 a, 1100 b, and 1100 c. An exemplary embodiment in which threecamera modules 1100 a, 1100 b, and 1100 c are disposed is illustrated inFIG. 12 , but the inventive concept is not limited thereto. In exemplaryembodiments of the inventive concept, the camera module group 1100 maybe modified to include only two camera modules. Additionally, inexemplary embodiments of the inventive concept, the camera module group1100 may be modified to include “n” camera modules (where n is a naturalnumber of 4 or more). In an exemplary embodiment of the inventiveconcept, each of the plurality of camera modules 1100 a, 1100 b, and1100 c of the camera module group 1100 may include the electronic device100 of FIG. 1 .

Below, a detailed configuration of the camera module 1100 b will be morefully described with reference to FIG. 13 , but the followingdescription may be equally applied to the remaining camera modules 1100a and 1100 c.

Referring to FIG. 13 , the camera module 1100 b may include a prism1105, an optical path folding element (OPFE) 1110, an actuator 1130, animage sensing device 1140, and a storage unit 1150.

The prism 1105 may include a reflecting plane 1107 of a light reflectingmaterial, and may change a path of a light “L” incident from theoutside.

In exemplary embodiments of the inventive concept, the prism 1105 maychange a path of the light “L” incident in a first direction “X” to asecond direction “Y” perpendicular to the first direction “X”,Additionally, the prism 1105 may change the path of the light “L”incident in the first direction “X” to the second direction “Y”perpendicular to the first direction “X” by rotating the reflectingplane 1107 of the light reflecting material in a direction “A” about acentral axis 1106 or rotating the central axis 1106 in a direction “B”.In this case, the OPPE, 1110 may move in a third direction “Z”perpendicular to the first direction “X” and the second direction “Y”.

In exemplary embodiments of the inventive concept, as illustrated, amaximum rotation angle of the prism 1105 in the direction “A” may beless than or equal to about 15 degrees in a positive A direction, andmay be greater than about 15 degrees in a negative A direction, but theinventive concept is not limited thereto.

In exemplary embodiments of the inventive concept, the prism 1105 maymove within approximately 20 degrees in a positive or negative Bdirection, between approximately 10 degrees and approximately 20degrees, or between approximately 15 degrees and approximately 20degrees; here, the prism 1105 may move at the same angle in the positiveor negative B direction or may move at a similar angle withinapproximately 1 degree.

In exemplary embodiments of the inventive concept, the prism 1105 maymove the reflecting plane 1107 of the light reflecting material in thethird direction (e.g., the Z direction) substantially parallel to adirection in which the central axis 1106 extends.

The OPFE 1110 may include optical lenses composed of “m” groups (where mis a natural number), for example. Here, “m” optical lenses may move inthe second direction “Y” to change an optical zoom ratio of the cameramodule 1100 b. For example, when a default optical zoom ratio of thecamera module 1100 b is “Z”, the optical zoom ratio of the camera module1100 b may be changed to an optical zoom ratio of 3Z, 5Z, or 5Z or more,by moving the “m” optical lenses included in the OPFE 1110.

The actuator 1130 may move the OPFE 1110 or an optical lens (hereinafterreferred to as an “optical lens”) to a specific location. For example,the actuator 1130 may adjust a location of an optical lens such that animage sensor 1142 is placed at a focal length of the optical lens foraccurate sensing.

The image sensing device 1140 may include the image sensor 1142, controllogic 1144, and a memory 1146. The image sensor 1142 may sense an imageof a sensing target by using the light “L” provided through an opticallens. The control logic 1144 may control overall operations of thecamera module 1100 b. For example, the control logic 1144 may control anoperation of the camera module 1100 b based on a control signal providedthrough a control signal line CSLb.

The memory 1146 may store information for an operation of the cameramodule 1100 b, such as calibration data 1147. The calibration data 1147may include information for the camera module 1100 b to generate imagedata by using the light “L” provided from the outside. The calibrationdata 1147 may include, for example, information about the degree ofrotation described above, information about a focal length, informationabout an optical axis, etc. In the case where the camera module 1100 bis implemented in the form of a multi-state camera in which a focallength varies depending on a location of an optical lens, thecalibration data 1147 may include a focal length value for each location(or state) of the optical lens and information about auto focusing.

The storage unit 1150 may store image data sensed through the imagesensor 1142. The storage unit 1150 may be disposed outside the imagesensing device 1140 and may be implemented in a shape where the storageunit 1150 and a sensor chip constituting the image sensing device 1140are stacked. In exemplary embodiments of the inventive concept, thestorage unit 1150 may be implemented with an electrically erasableprogrammable read only memory (EEPROM), but the inventive concept is notlimited thereto. Referring to FIGS. 12 and 13 together, in exemplaryembodiments of the inventive concept, each of the plurality of cameramodules 1100 a, 1100 b, and 1100 c may include the actuator 1130. Assuch, the same calibration data 1147 or different calibration data 1147may be included in the plurality of camera modules 1100 a, 1100 b, and1100 c depending on operations of the actuators 1130 therein.

In exemplary embodiments of the inventive concept, one camera module(e.g., 1100 b) among the plurality of camera modules 1100 a, 1100 b, and1100 c may be a camera module having a folded lens shape in which theprism 1105 and the OPFE 1110 described above are included, and theremaining camera modules (e.g., 1100 a and 1100 c) may each be a cameramodule having a vertical shape in which the prism 1105 and the OPFE 1110described above are not included; however, the inventive concept is notlimited thereto.

In exemplary embodiments of the inventive concept, one camera module(e.g., 1100 c) among the plurality of camera modules 1100 a, 1100 b, and1100 c may be, for example, depth camera having a vertical shapeextracting depth information by using an infrared ray (IR). In thiscase, the application processor 1200 may merge image data provided fromthe depth camera and image data provided from any other camera module(e.g., 1100 a or 1100 b), and may generate a three-dimensional (3D)depth image.

In exemplary embodiments of the inventive concept, at least two cameramodules (e.g., 1100 a and 1100 b) among the plurality of camera modules1100 a, 1100 b, and 1100 c may have different fields of view. In thiscase, the at least two camera modules (e.g., 1100 a and 1100 b) amongthe plurality of camera modules 1100 a, 1100 b, and 1100 c may includedifferent optical lens, but the inventive concept is not limitedthereto.

Additionally, in exemplary embodiments of the inventive concept, fieldsof view of the plurality of camera modules 1100 a, 1100 b, and 1100 cmay be different. In this case, the plurality of camera modules 1100 a,1100 b, and 1100 c may include different optical lens, but the inventiveconcept is not limited thereto.

In exemplary embodiments of the inventive concept, the plurality ofcamera modules 1100 a, 1100 b, and 1100 c may be disposed to bephysically separated from one another. In other words, the plurality ofcamera modules 1100 a, 1100 b, and 1100 c may not use a sensing area ofone image sensor 1142, but the plurality of camera modules 1100 a, 1100b, and 1100 c may include independent image sensors 1142 therein,respectively.

Returning to FIG. 12 , the application processor 1200 may include animage processing device 1210, a memory controller 1220, and an internalmemory 1230. The application processor 1200 may be implemented to beseparated from the plurality of camera modules 1100 a, 1100 b, and 1100c. For example, the application processor 1200 and the plurality ofcamera modules 1100 a, 1100 b, and 1100 c may be implemented withseparate semiconductor chips.

The image processing device 1210 may include a plurality of sub imageprocessors 1212 a, 1212 b, and 1212 c, an image generator 1214, and acamera module controller 1216.

The image processing device 1210 may include the plurality of sub imageprocessors 1212 a, 1212 b, and 1212 c, the number of which correspondsto the number of the plurality of camera modules 1100 a, 1100 b, and1100 c.

Image data generated from the camera modules 1100 a, 1100 b, and 1100 cmay be respectively provided to the corresponding sub image processors1212 a, 1212 b, and 1212 c through separated image signal lines ISLa,ISLb, and ISLc. For example, the image data generated from the cameramodule 1100 a may be provided to the sub image processor 1212 a throughthe image signal line ISLa, the image data generated from the cameramodule 1100 b may be provided to the sub image processor 1212 b throughthe image signal line ISLb, and the image data generated from the cameramodule 1100 c may be provided to the sub image processor 1212 c throughthe image signal line ISLc. This image data transmission may beperformed, for example, by using a camera serial interface (CSI) basedon the MIPI (Mobile Industry Processor Interface) specification, but theinventive concept is not limited thereto.

Meanwhile, in exemplary embodiments of the inventive concept, one subimage processor may be disposed to correspond to a plurality of cameramodules. For example, the sub image processor 1212 a and the sub imageprocessor 1212 c may be integrally implemented, not separated from eachother as illustrated in FIG. 12 ; in this case, one of the pieces ofimage data respectively provided from the camera module 1100 a and thecamera module 1100 c may be selected through a selection element (e.g.,a multiplexer), and the selected image data may be provided to theintegrated sub image processor.

The image data respectively provided to the sub image processors 1212 a,1212 b, and 1212 c may be provided to the image generator 1214. Theimage generator 1214 may generate an output image by using the imagedata respectively provided from the sub image processors 1212 a, 1212 b,and 1212 c, depending on generating information (or image generatinginformation) or a mode signal.

In detail, the image generator 1214 may generate the output image bymerging at least a portion of the image data respectively generated fromthe camera modules 1100 a, 1100 b, and 1100 c having different fields ofview, depending on the generating information or the mode signal.Additionally, the image generator 1214 may generate the output image byselecting one of the image data respectively generated from the cameramodules 1100 a, 1100 b, and 1100 c having different fields of view,depending on the generating information or the mode signal.

In exemplary embodiments of the inventive concept, the generatinginformation may include a zoom signal or a zoom factor. Additionally, inexemplary embodiments of the inventive concept, the mode signal may be,for example, a signal based on a mode selected from a user.

In the case where the generating information is the zoom signal (or zoomfactor) and the camera modules 1100 a, 1100 b, and 1100 c have differentvisual fields (or fields of view), the image generator 1214 may performdifferent operations depending on a kind of the zoom signal. Forexample, in the case where the zoom signal is a first signal, the imagegenerator 1214 may merge the image data output from the camera module1100 a and the image data output from the camera module 1100 c, and maygenerate the output image by using the merged image signal and the imagedata output from the camera module 1100 b that is not used in themerging operation. In the case where the zoom signal is a second signaldifferent from the first signal, without the image data mergingoperation, the image generator 1214 may select one of the image datarespectively output from the camera modules 1100 a, 1100 b, and 1100 c,and may output the selected image data as the output image. However, theinventive concept is not limited thereto, and a way to process imagedata may be modified without limitation if necessary.

In exemplary embodiments of the inventive concept, the image generator1214 may generate merged image data having an increased dynamic range byreceiving a plurality of image data of different exposure times from atleast one of the plurality of sub image processors 1212 a, 1212 b, and1212 c, and performing high dynamic range (HDR) processing on theplurality of image data.

The camera module controller 1216 may provide control signals to thecamera modules 1100 a, 1100 b, and 1100 c. The control signals generatedfrom the camera module controller 1216 may be respectively provided tothe corresponding camera modules 1100 a, 1100 b, and 1100 c throughcontrol signal lines CSLa, CSLb, and CSLc separated from one another.

One of the plurality of camera modules 1100 a, 1100 b, and 1100 c may bedesignated as a master camera (e.g., 1100 b) depending on the generatinginformation including a zoom signal or the mode signal, and theremaining camera modules (e.g., 1100 a and 1100 c) may be designated asa slave camera. The above designation information may be included in thecontrol signals, and the control signals including the designationinformation may be respectively provided to the corresponding cameramodules 1100 a, 1100 b, and 1100 c through the control signal linesCSLa, CSLb, and CSLc separated from one another.

Camera modules operating as a master and a slave may be changeddepending on the zoom factor or an operating mode signal. For example,in the case where the field of view of the camera module 1100 a is widerthan the field of view of the camera module 1100 b and the zoom factorindicates a low zoom ratio, the camera module 1100 b may operate as amaster, and the camera module 1100 a may operate as a slave. Incontrast, in the case where the zoom factor indicates a high zoom ratio,the camera module 1100 a may operate as a master, and the camera module1100 b may operate as a slave.

In exemplary embodiments of the inventive concept, the control signalprovided from the camera module controller 1216 to each of the cameramodules 1100 a, 1100 b, and 1100 c may include a sync enable signal. Forexample, in the case where the camera module 1100 b is used as a mastercamera and the camera modules 1100 a and 1100 c are used as a slavecamera, the camera module controller 1216 may transmit the sync enablesignal to the camera module 1100 b. The camera module 1100 b that isprovided with the sync enable signal may generate a sync signal based onthe provided sync enable signal, and may provide the generated syncsignal to the camera modules 1100 a and 1100 c through a sync signalline SSL. The camera module 1100 b and the camera modules 1100 a and1100 c may be synchronized with the sync signal to transmit image datato the application processor 1200.

In exemplary embodiments of the inventive concept, the control signalprovided from the camera module controller 1216 to each of the cameramodules 1100 a, 1100 b, and 1100 c may include mode informationaccording to the mode signal. Based on the mode information, theplurality of camera modules 1100 a, 1100 b, and 1100 c may operate in afirst operating mode and a second operating mode with regard to asensing speed.

In the first operating mode, the plurality of camera modules 1100 a,1100 b, and 1100 c may generate image signals at a first speed (e.g.,may generate image signals at a first frame rate), may encode the imagesignals at a second speed (e.g., may encode the image signal at a secondframe rate higher than the first frame rate), and transmit the encodedimage signals to the application processor 1200. In this case, thesecond speed may be about 30 times or less the first speed.

The application processor 1200 may store the received image signals,e.g., the encoded image signals, in the internal memory 1230 providedtherein or the external memory 1400 provided outside the applicationprocessor 1200. Afterwards, the application processor 1200 may read anddecode the encoded image signals from the internal memory 1230 or theexternal memory 1400, and may display image data generated based on thedecoded image signals. For example, a corresponding one among sub imageprocessors 1212 a, 1212 b, and 1212 c of the image processing device1210 may perform decoding and may also perform image processing on thedecoded image signal.

In the second operating mode, the plurality of camera modules 1100 a,1100 b, and 1100 c may generate image signals at a third speed (e.g.,may generate image signals at a third frame rate lower than the firstframe rate), and transmit the image signals to the application processor1200. The image signals provided to the application processor 1200 maybe signals that are not encoded. The application processor 1200 mayperform image processing on the received image signals or may store theimage signals in the internal memory 1230 or the external memory 1400.

The PMIC 1300 may supply powers, for example, power supply voltages, tothe plurality of camera modules 1100 a, 1100 b, and 1100 c. For example,under control of the application processor 1200, the PMIC 1300 maysupply a first power to the camera module 1100 a through a power signalline PSLa, may supply a second power to the camera module 1100 b througha power signal line PSLb, and may supply a third power to the cameramodule 1100 c through a power signal line PSLc.

In response to a power control signal PCON from the applicationprocessor 1200, the PMIC 1300 may generate a power corresponding to eachof the plurality of camera modules 1100 a, 1100 b, and 1100 c, and mayadjust a level of the power. The power control signal PCON may include apower adjustment signal for each operating mode of the plurality ofcamera modules 1100 a, 1100 b, and 1100 c. For example, the operatingmodes may include a low-power mode. In this case, the power controlsignal PCON may include information about a camera module operating inthe low-power mode and a set power level. Levels of the powersrespectively provided to the plurality of camera modules 1100 a, 1100 b,and 1100 c may be identical to each other or may be different from oneanother. Additionally, a level of a power may be dynamically changed.

The camera module 1100 b may correspond to the electronic device 100 ofFIG. 1 . The image sensing device 1140 may correspond to the imagesensor 110 of FIG. 1 . The storage unit 1150 may include the processor120 of FIG. 1 .

In the above exemplary embodiments, the terms “image data”, “partialimage data”, “partial data”, and “pieces of partial data” are used. Theterms may be interchangeably used without departing from the spirit andscope of the inventive concept. In an exemplary embodiment of theinventive concept, image data may indicate data corresponding to oneframe, and the meaning of the image data described above is not limited.The terms “partial image data”, “partial data”, and “pieces of partialdata” may indicate a portion of data of one frame, but meanings of“partial image data”, “partial data”, and “pieces of partial data” arenot limited.

In the above description, components according to exemplary embodimentsof the inventive concept are described by using the terms “first”,“second”, “third”, and the like. However, the terms “first”, “second”,“third”, and the like may be used to distinguish components from oneanother and do not limit the inventive concept. For example, the terms“first”, “second”, “third”, and the like do not indicate an order or anumerical meaning of any form.

In the above description, components according to exemplary embodimentsof the inventive concept are described by using blocks. The blocks maybe implemented with various hardware devices, such as an integratedcircuit (IC), an application-specific IC (ASCI), a field programmablegate array (FPGA), or a complex programmable logic device (CPLD),firmware driven in hardware devices, software such as an application, ora combination of a hardware device and software. Additionally, theblocks may include circuits implemented with semiconductor elements inan integrated circuit or circuits enrolled/protected as intellectualproperty (IP).

According to exemplary embodiments of the inventive concept, high-powerand high-quality conversion and low-power and low-quality conversion areselectively performed depending on a resolution or an image quality ofportions of image data obtained by an image sensor. Accordingly, anelectronic device capable of generating image data having an improvedresolution or image quality and reducing power consumption, as well asan operating method of the electronic device, are provided.

While the inventive concept has been described with reference toexemplary embodiments thereof, it will be apparent to those of ordinaryskill in the art that various changes and modifications in form anddetails may be made thereto without departing from the spirit and scopeof the inventive concept as set forth in the following claims.

What is claimed is:
 1. An electronic device comprising: an image sensorconfigured to capture a target to generate first image data; and aprocessor, wherein the processor is configured to: perform directionalinterpolation on a first area of the first image data to generate firstpartial image data; perform upscale on a second area of the first imagedata to generate second partial image data; perform the directionalinterpolation on a third area of the first image data; perform theupscale on the third area of the first image data; and combine the firstpartial image data and the second partial image data to generate secondimage data, wherein the first area and the second area are different,and wherein the directional interpolation generates image data having aresolution higher than that of image data generated by the upscale. 2.The electronic device of claim 1, wherein the first image data is basedon a non-Bayer pattern, and wherein the second image data is based on aBayer pattern.
 3. The electronic device of claim 1, wherein theprocessor is further configured to: perform alpha blending on a resultof the directional interpolation on the third area of the first imagedata and a result of the upscale on the third area of the first imagedata to generate third partial image data.
 4. The electronic device ofclaim 3, wherein the processor is further configured to: combine thefirst partial image data, the second partial image data, and the thirdpartial image data to generate the second image data.
 5. The electronicdevice of claim 3, wherein the third area is divided into two or moresub-areas, and wherein the processor differently adjusts alpha values tobe applied to results of the directional interpolation and results ofthe upscale in the two or more sub-areas.
 6. The electronic device ofclaim 5, wherein the first area includes a circular area placed at acenter of the first image data, and wherein the two or more sub-areasinclude two or more concentric circles surrounding the first area. 7.The electronic device of claim 1, wherein the first area includes acircular area placed at a center of the first image data, and whereinthe second area includes a remaining area of the first image data, whichsurrounds the first area.
 8. The electronic device of claim 1, whereinthe processor determines the first area and the second area usinglocation information obtained in a process of calibrating the imagesensor.
 9. The electronic device of claim 8, wherein the locationinformation indicates the first area and the second area based on adistance from a center of the first image data.
 10. The electronicdevice of claim 9, wherein the location information is based on a changeof a modulation transfer function (MTF) according to a distance from thecenter of the first image data.
 11. An operating method of an electronicdevice which includes an image sensor and a processor, the operatingmethod comprising: capturing, at the image sensor, a target to generatefirst image data; performing, at the processor, directionalinterpolation on a first area of the first image data to generate firstpartial image data; performing, at the processor, upscale on a secondarea of the first image data to generate second partial image data;performing, at the processor, the directional interpolation on a thirdarea of the first image data to generate third partial image data;performing, at the processor, the upscale on the third area of the firstimage data to generate fourth partial image data; and combining, at theprocessor, the first partial image data and the second partial imagedata to generate second image data.
 12. The operating method of claim11, further comprising: performing, at the processor, alpha blending onthe third partial image data and the fourth partial image data togenerate fifth partial image data.
 13. The operating method of claim 12,wherein the combining of the first partial image data and the secondpartial image data to generate the second image data includes:combining, at the processor, the first partial image data, the secondpartial image data, and the fifth partial image data to generate thesecond image data.
 14. The operating method of claim 12, wherein theperforming of the alpha blending to generate the fifth partial imagedata includes: selecting the fourth partial image data as backgroundimage data; selecting the third partial image data as foreground imagedata; and adjusting an alpha value of the third partial image data. 15.The operating method of claim 12, further comprising: performing, at theprocessor, the directional interpolation on a fourth area of the firstimage data to generate sixth partial image data; performing, at theprocessor, the upscale on the fourth area of the first image data togenerate seventh partial image data; and performing, at the processor,alpha blending on the sixth partial image data and the seventh partialimage data to generate eighth partial image data, wherein an alpha valueapplied to the fifth partial image data and an alpha value applied tothe eighth partial image data are different.
 16. The operating method ofclaim 12, wherein the first image data is converted into the secondimage data using the first partial image data, the second partial imagedata, and the fifth partial image data, in units of frames.
 17. Anelectronic device comprising: an image sensor; and a processorconfigured to receive first image data from the image sensor and toconvert and output the first image data as second image data, whereinthe image sensor includes: a lens; a color filter array including colorfilters configured to pass specific frequency components of a lightincident through the lens; a pixel array including pixels configured toconvert intensities of the specific frequency components of the lightpassing through the color filters into analog signals; and ananalog-to-digital converter configured to convert the analog signalsinto digital signals and to output the digital signals to the imagesensor, wherein the processor includes: a first memory configured toreceive the first image data; location information storage configured tostore location information, wherein the location information includes afirst area having a resolution level higher than a first threshold leveland a second area having a resolution level lower than or equal to asecond threshold level; a first conversion circuit configured to performa first conversion on first input data and to output a result of thefirst conversion as first partial image data; a second conversioncircuit configured to perform a second conversion on second input dataand to output a result of the second conversion as second partial imagedata; a selection circuit configured to output the first image data in aform of the first input data, the second input data, or the first inputdata and the second input data, based on the location information; and amixer configured to combine the first partial image data and the secondpartial image data to generate the second image data, in response to aselection signal from the selection circuit, and wherein, when the firstpartial image data and the second partial image data are outputtogether, the mixer is configured to perform alpha blending on the firstpartial image data and the second partial image data.
 18. The electronicdevice of claim 17, wherein the first conversion is directionalinterpolation, and wherein the second conversion is upscale.